Papers with attention layers
Adversarial Attention Modeling for Multi-dimensional Emotion Regression (P19-1)
Copied to clipboard
| Challenge: | Empirical evaluation on EMOBANK corpus shows that our approach achieves notable improvements in r-values over the state-of-the-art baselines. |
| Approach: | They propose a neural network-based approach to multi-dimensional emotion regression which automatically rates multiple emotion dimension scores for an input text. |
| Outcome: | The proposed approach achieves notable improvements in r-values on both EMOBANK Reader’s and Writer’s multi-dimensional emotion regression tasks over the state-of-the-art baselines. |
White-box Testing of NLP models with Mask Neuron Coverage (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent research has shown that black-box testing is not applicable to NLP models. |
| Approach: | They propose a set of white-box testing methods that are customized for transformer-based NLP models and adapt them to a black-box test suite. |
| Outcome: | The proposed methods can reduce testing suites by 60% while retaining failing tests, thereby concentrating faultdetection power of the test suite. |
Self-Detoxifying Language Models via Toxification Reversal (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate toxic content in pretrained language models are resource-intensive and require additional components. |
| Approach: | They propose a method that enables the PLM itself to achieve "self-detoxification" they identify the toxification direction from the normal generation process to the one prompted with the negative prefix and then steer the generation to the reverse direction by manipulating the information movement within the attention layers. |
| Outcome: | The proposed method can achieve comparable performance with state-of-the-art methods without any fine-tuning or extra components. |
Is Attention Interpretable? (P19-1)
Copied to clipboard
| Challenge: | Attention mechanisms have recently boosted performance on a range of NLP tasks. |
| Approach: | They propose to manipulate attention weights in text classification models and analyze the resulting differences in their predictions. |
| Outcome: | The proposed approach improves models' predictions by using gradient-based rankings of attention weights. |
Modeling Context With Linear Attention for Scalable Document-Level Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level machine translation models lack quadratic complexity in the sequence length due to their attention layers. |
| Approach: | They evaluate a recent linear attention model with a sentential gate to promote a recency inductive bias and compare it to open-source document translation. |
| Outcome: | The proposed model significantly improves translation quality on IWSLT 2015 and OpenSubtitles 2018 with similar or better BLEU scores. |
Structured Context and High-Coverage Grammar for Conversational Question Answering over Knowledge Graphs (2021.emnlp-main)
Copied to clipboard
| Challenge: | We present a new approach for weakly-supervised conversational Question Answering over Knowledge Graphs . |
| Approach: | They propose a Logical Form grammar that can model a wide range of queries on a Knowledge Graph while remaining sufficiently simple to generate supervision data efficiently. |
| Outcome: | The proposed grammar can model a wide range of queries while remaining simple to generate supervision data efficiently. |